Skip to content

Dependence aware tests for ppc*ecdf#428

Open
florence-bockting wants to merge 152 commits intostan-dev:masterfrom
florence-bockting:dependence-aware-LOO-PIT
Open

Dependence aware tests for ppc*ecdf#428
florence-bockting wants to merge 152 commits intostan-dev:masterfrom
florence-bockting:dependence-aware-LOO-PIT

Conversation

@florence-bockting
Copy link
Copy Markdown

@florence-bockting florence-bockting commented Mar 4, 2026 •
edited
Loading

Description

The current approach in ppc_loo_pit_ecdf and ppc_pit_ecdf assumes independence of LOO-PIT values which is not valid (Marhunenda et al., 2005). The corresponding graphical test yields an envelope that is too wide, reducing the test's ability to reveal model miscalibration.
Tesso & Vehtari (2026, see preprint) propose three testing procedures that can handle any dependent uniform values and provide an updated graphical representation that uses color coding to indicate influential regions or most influential points of the ECDF. This PR implements the new development, by adding the updated approach (method = "correlated") additionally to the previous approach (method = "independent").

TODOs

  • updated ppc_loo_pit_ecdf() function in ppc-loo.R
  • updated ppc_pit_ecdf() and ppc_pit_ecdf_grouped() function in ppc-distributions.R
  • add unittests and visual regression tests
  • update documentation
  • deprecation suggestion of old method
    • P1: old method (default) and new method available via method argument
    • P2: new method (default) and old method available via method argument
    • P3: old method is removed

@florence-bockting
add helper functions for correlation-aware uniformity test
3e0e568
@florence-bockting
incorporate in ppc_loo_pit_ecdf correlation-aware uniformity test
c02215c
@florence-bockting
add tests for updated ppc_loo_pit_ecdf function
0d58176
@florence-bockting
formatting and efficiency improvements
ee2a3fc
@florence-bockting
improve description and computation of uniformity tests for pcc_loo_p…
583b3e2 
…it_ecdf
@florence-bockting
add unit-tests for dependence-aware uniformity tests
317a753
@florence-bockting
do not remove NA when using sort() in pot_test
3afa4ac
@florence-bockting
move unittest for uniformity tests from test-ppc-loo.R to test-helper…
c8e63fd 
…s-ppc.R
@florence-bockting
improve descripting and naming of function to compute shapley values
609f48f
@florence-bockting
improve descripting and naming of function to compute shapley values
bbfedc5
@florence-bockting
add unittest for compute_shapley_values
18f6539
@florence-bockting
improve computation of Cauchy combination test and std. Cauchy values
2ca91a3
@florence-bockting
make variable namings more descriptive
d6a18ce
@florence-bockting
add unittests for Cauchy combination test
8b55c2f
@florence-bockting
improve efficiency of influential_points_idx
da23cdf
@florence-bockting
add unittests for influential_points_idx
32a4087
@florence-bockting
update unittests for ppc_loo_pit_ecdf
7ca5b9b
@florence-bockting
remove influential_points_idx function
a66358f
@florence-bockting
remove unittests for influential_points_idx
d62cd06
@florence-bockting
remove infl_points_only argument, add linewith, color arguments, and …
753053f 
…improve input validation checks
@florence-bockting
minor fixes in plotting behavior (fontsize, linestyle) and input vali…
1ae2595 
…dation
@florence-bockting
adjust tests to new ppc_loo_pit_ecdf implementation
270f67e
@florence-bockting
add compute_cauchy function
7489701
@florence-bockting
transition from -qcauchy(x) to compute_cauchy(x)
22048bd
@florence-bockting
add unittests for compute_cauchy
8180cb7
@florence-bockting
minor adjustments for regression-tests
703b9f0
@florence-bockting
change svg snapshots for regression tests
7f6fb12
@florence-bockting
improve documentation for arguments linewidth, color in ppc_loo_pit_ecdf
11086f9
@florence-bockting
update PPC-loo.Rd file with new arguments
5321ac0
@florence-bockting
add tests for plot_diff=TRUE and method=correlated
478e96e
@florence-bockting florence-bockting mentioned this pull request Mar 26, 2026
Open
@florence-bockting florence-bockting changed the title Dependence aware tests for ppc_loo_pit_ecdf Dependence aware tests for ppc_*_ecdf Apr 5, 2026
@florence-bockting florence-bockting changed the title Dependence aware tests for ppc_*_ecdf Dependence aware tests for ppc*ecdf Apr 5, 2026
@florence-bockting
Copy link
Copy Markdown
Author

florence-bockting commented Apr 7, 2026

As mentioned in the PR introduction text, the idea would be migration in multiple stages.
First stage is non-breaking behavior: That is users can simply use "original" code but are informed that this behavior will change in future. Future changes consider changing the default method and thus are breaking.

Informing the user in the first stage is currently implemented as follows:

# using the "original" code provides a message about the change
> ppc_loo_pit_ecdf(y, yrep, lw)
ℹ In the next major release, the default `method` will change to 'correlated'.
• To silence this message, explicitly set `method = 'independent'` or `method = 'correlated'`.

# explicit use of "independent" method provides information that this method is superseded
> ppc_loo_pit_ecdf(y, yrep, lw, method = "independent")
The 'independent' method is superseded by the 'correlated' method.

@florence-bockting florence-bockting marked this pull request as ready for review April 7, 2026 05:46
@florence-bockting florence-bockting mentioned this pull request Apr 7, 2026
Draft
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avehtari avehtari Awaiting requested review from avehtari

@jgabry jgabry Awaiting requested review from jgabry

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@florence-bockting @codecov-commenter @jgabry